{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "ganivut.ipynb",
      "private_outputs": true,
      "provenance": [],
      "machine_shape": "hm",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU",
    "gpuClass": "premium"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/eyaler/avatars4all/blob/master/ganivut.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MQTh8Pl_w4MK"
      },
      "source": [
        "# Demo for paper \"Liquid Warping GAN with Attention: A Unified Framework for Human Image Synthesis (Impersonator++)\"\n",
        "\n",
        "### Made just a little bit more accessible by Eyal Gruss (https://eyalgruss.com, eyalgruss@gmail.com)\n",
        "\n",
        "##### Original project: https://www.impersonator.org/work/impersonator-plus-plus.html\n",
        "\n",
        "##### Original notebook: <https://colab.research.google.com/drive/1bwUnj-9NnJA2EMr7eWO4I45UuBtKudg_>\n",
        "\n",
        "#### **Stuff I made**:\n",
        "##### Avatars4all repository: https://github.com/eyaler/avatars4all\n",
        "##### Notebook for live webcam in the browser: https://colab.research.google.com/github/eyaler/avatars4all/blob/master/fomm_live.ipynb\n",
        "##### Notebook for talking head model: https://colab.research.google.com/github/eyaler/avatars4all/blob/master/fomm_bibi.ipynb\n",
        "##### Notebook for full body models (FOMM): https://colab.research.google.com/github/eyaler/avatars4all/blob/master/fomm_fufu.ipynb\n",
        "##### Notebook for full body models (impersonator): https://colab.research.google.com/github/eyaler/avatars4all/blob/master/ganozli.ipynb\n",
        "##### Notebook for full body models (impersonator++): https://colab.research.google.com/github/eyaler/avatars4all/blob/master/ganivut.ipynb\n",
        "##### Notebook for Wav2Lip audio based lip syncing: https://colab.research.google.com/github/eyaler/avatars4all/blob/master/melaflefon.ipynb\n",
        "##### List of more generative tools: https://j.mp/generativetools"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q45P7Uicpsuc",
        "cellView": "form"
      },
      "source": [
        "#@title Setup\n",
        "%cd /content\n",
        "!git clone --depth 1 https://github.com/eyaler/iPERCore\n",
        "\n",
        "!apt-get -o Dpkg::Options::=\"--force-overwrite\" install cuda-11-1\n",
        "!apt-get install ffmpeg\n",
        "import os\n",
        "cuda_home = '/usr/local/cuda-11.1'\n",
        "os.environ[\"CUDA_HOME\"] = cuda_home\n",
        "with open(os.path.join(cuda_home, 'version.txt'), 'w') as f:\n",
        "  f.write('CUDA Version 11.1')\n",
        "\n",
        "%cd /content/iPERCore\n",
        "!python setup.py develop\n",
        "%cd /content\n",
        "\n",
        "!wget -nc -O /content/iPERCore/assets/checkpoints.zip \"http://101.32.75.151:10086/checkpoints.zip\"\n",
        "!wget -nc -O /content/iPERCore/assets/checkpoints.zip \"https://eyalgruss.com/fomm/checkpoints.zip\"\n",
        "!unzip -u /content/iPERCore/assets/checkpoints.zip -d /content/iPERCore/assets/\n",
        "\n",
        "#!wget -nc -O /content/iPERCore/assets/samples.zip  \"http://101.32.75.151:10086/samples.zip\"\n",
        "#!wget -nc -O /content/iPERCore/assets/samples.zip  \"https://eyalgruss.com/fomm/samples.zip\"\n",
        "#!unzip -u /content/iPERCore/assets/samples.zip -d /content/iPERCore/assets/\n",
        "\n",
        "!pip install -U git+https://github.com/ytdl-org/youtube-dl\n",
        "!pip install imageio==2.9.0\n",
        "!pip install imageio-ffmpeg==0.4.5\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VEnzredFvnwJ",
        "cellView": "form"
      },
      "source": [
        "#@title Get the Driver video, Avatar and Background images from the web\n",
        "#@markdown 1. You can change the URLs to your **own** stuff!\n",
        "#@markdown 2. Alternatively, you can upload **local** files in the next cells\n",
        " \n",
        "video_url = 'https://www.youtube.com/watch?v=NdEPmitJvaA' #@param {type:\"string\"}\n",
        "image_url = 'https://i.pinimg.com/564x/74/9e/96/749e96293399ebe1b8c99d89be522383.jpg' #@param {type:\"string\"}\n",
        "background_url = 'https://c.pxhere.com/photos/a9/9a/forest_tree_trees_foliage_leaves_leaf_nature_needles-1005931.jpg!d' #@param {type:\"string\"}\n",
        "\n",
        "if video_url:\n",
        "  !rm -f /content/video.mp4\n",
        "  !youtube-dl --no-check-extensions --no-playlist -f \"bestvideo[ext=mp4][vcodec!*=av01][height<=720]+bestaudio[ext=m4a]/mp4[height<=720][vcodec!*=av01]/mp4[vcodec!*=av01]/mp4\" \"$video_url\" --merge-output-format mp4 -o /content/video\n",
        "  !mv /content/video.mp4 /content/video \n",
        " \n",
        "if image_url:\n",
        "  !wget \"$image_url\" -O /content/image\n",
        "\n",
        "if background_url:\n",
        "  !wget \"$background_url\" -O /content/background"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4L4lDMont6aP",
        "cellView": "form"
      },
      "source": [
        "#@title Optionally upload local Driver video { run: \"auto\" }\n",
        "manually_upload_video = False #@param {type:\"boolean\"}\n",
        "if manually_upload_video:\n",
        "  from google.colab import files\n",
        "  import shutil\n",
        "\n",
        "  %cd /content/sample_data\n",
        "  try:\n",
        "    uploaded = files.upload()\n",
        "  except Exception as e:\n",
        "    %cd /content\n",
        "    raise e\n",
        "\n",
        "  for fn in uploaded:\n",
        "    shutil.move('/content/sample_data/'+fn, '/content/video')\n",
        "    break\n",
        "  %cd /content"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ExeY9p-Ht-_6",
        "cellView": "form"
      },
      "source": [
        "#@title Optionally upload local Avatar image { run: \"auto\" }\n",
        "manually_upload_image = False #@param {type:\"boolean\"}\n",
        "if manually_upload_image:\n",
        "  from google.colab import files\n",
        "  import shutil\n",
        " \n",
        "  %cd /content/sample_data\n",
        "  try:\n",
        "    uploaded = files.upload()\n",
        "  except Exception as e:\n",
        "    %cd /content\n",
        "    raise e\n",
        "\n",
        "  for fn in uploaded:\n",
        "    shutil.move('/content/sample_data/'+fn, '/content/image')\n",
        "    break\n",
        "  %cd /content"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "7b6UW1KM2Myw"
      },
      "source": [
        "#@title Optionally upload local Background image { run: \"auto\" }\n",
        "manually_upload_background = False #@param {type:\"boolean\"}\n",
        "if manually_upload_background:\n",
        "  from google.colab import files\n",
        "  import shutil\n",
        " \n",
        "  %cd /content/sample_data\n",
        "  try:\n",
        "    uploaded = files.upload()\n",
        "  except Exception as e:\n",
        "    %cd /content\n",
        "    raise e\n",
        "\n",
        "  for fn in uploaded:\n",
        "    shutil.move('/content/sample_data/'+fn, '/content/background')\n",
        "    break\n",
        "  %cd /content"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eUBfTA8YuEAy",
        "cellView": "form"
      },
      "source": [
        "#@title Optionally shorten Driver video\n",
        "start_seconds = 0 #@param {type:\"number\"}\n",
        "duration_seconds =  60#@param {type:\"number\"}\n",
        "start_seconds = max(start_seconds,0)\n",
        "duration_seconds = max(duration_seconds,0)\n",
        "\n",
        "if duration_seconds: \n",
        "  !mv /content/video /content/full_video\n",
        "  !ffmpeg -ss $start_seconds -t $duration_seconds -i /content/full_video -f mp4 /content/video -y"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cQ7R6KGHuEqI",
        "cellView": "form"
      },
      "source": [
        "#@title Prepare assets\n",
        "#@markdown If you ran out of RAM this means that the video is too large. You can shorten it above.\n",
        " \n",
        "#center_video_to_body = True #@param {type:\"boolean\"}\n",
        "#crop_video_to_body = True #@param {type:\"boolean\"}\n",
        "#video_crop_expansion_factor = 1.05 #@param {type:\"number\"}\n",
        "center_image_to_body = True #@param {type:\"boolean\"}\n",
        "crop_image_to_body = False #@param {type:\"boolean\"}\n",
        "image_crop_expansion_factor = 1.05 #@param {type:\"number\"}\n",
        "#video_crop_expansion_factor = max(video_crop_expansion_factor, 1)\n",
        "image_crop_expansion_factor = max(image_crop_expansion_factor, 1)\n",
        "keep_aspect_background = True #@param {type:\"boolean\"}\n",
        "image_size = 512\n",
        "\n",
        "import imageio\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import matplotlib.animation as animation\n",
        "from skimage.transform import resize\n",
        "from IPython.display import HTML, clear_output\n",
        "import cv2\n",
        "import shutil\n",
        "import os\n",
        "import warnings\n",
        "warnings.filterwarnings(\"ignore\")\n",
        " \n",
        "hog = cv2.HOGDescriptor()\n",
        "hog.setSVMDetector(cv2.HOGDescriptor_getDefaultPeopleDetector())\n",
        " \n",
        "def fix_dims(im):\n",
        "    if im.ndim == 2:\n",
        "        im = np.tile(im[..., None], [1, 1, 3])\n",
        "    return im[...,:3]\n",
        " \n",
        "def get_crop(im, center_body=True, crop_body=True, expansion_factor=1, rects=None):\n",
        "    im = fix_dims(im)\n",
        "    if (center_body or crop_body) and rects is None:\n",
        "        rects, _ = hog.detectMultiScale(im, winStride=(4, 4),padding=(8,8), scale=expansion_factor)\n",
        "    if (center_body or crop_body) and rects is not None and len(rects):\n",
        "        x0,y0,w,h = sorted(rects, key=lambda x: x[2]*x[3])[-1]\n",
        "        if crop_body:\n",
        "            x0 += w//2-h//2\n",
        "            x1 = x0+h\n",
        "            y1 = y0+h\n",
        "        else:\n",
        "            img_h,img_w = im.shape[:2]\n",
        "            x0 += (w-img_h)//2\n",
        "            x1 = x0+img_h\n",
        "            y0 = 0\n",
        "            y1 = img_h\n",
        "    else:\n",
        "        h,w = im.shape[:2]\n",
        "        x0 = (w-h)//2\n",
        "        x1 = (w+h)//2\n",
        "        y0 = 0\n",
        "        y1 = h\n",
        "    return int(x0),int(x1),int(y0),int(y1)\n",
        " \n",
        "def pad_crop_resize(im, x0=None, x1=None, y0=None, y1=None, new_h=image_size, new_w=image_size):\n",
        "    im = fix_dims(im)\n",
        "    h,w = im.shape[:2]\n",
        "    if x0 is None:\n",
        "      x0 = 0\n",
        "    if x1 is None:\n",
        "      x1 = w\n",
        "    if y0 is None:\n",
        "      y0 = 0\n",
        "    if y1 is None:\n",
        "      y1 = h\n",
        "    if x0<0 or x1>w or y0<0 or y1>h:\n",
        "        im = np.pad(im, pad_width=[(max(-y0,0),max(y1-h,0)),(max(-x0,0),max(x1-w,0)),(0,0)], mode='edge')\n",
        "    return resize(im[max(y0,0):y1-min(y0,0),max(x0,0):x1-min(x0,0)], (new_h, new_w))\n",
        "\n",
        "def crop_resize(im, size, crop=False):\n",
        "  if im.shape[:2] == size:\n",
        "    return im\n",
        "  if size[0]<im.shape[0] or size[1]<im.shape[1]:\n",
        "    interp = cv2.INTER_AREA\n",
        "  else:\n",
        "    interp = cv2.INTER_CUBIC\n",
        "  if not crop:\n",
        "    return np.clip(cv2.resize(im, size[::-1], interpolation=interp),0,1)\n",
        "  ratio = max(size[0]/im.shape[0], size[1]/im.shape[1])\n",
        "  im = np.clip(cv2.resize(im, (int(np.ceil(im.shape[1]*ratio)), int(np.ceil(im.shape[0]*ratio))), interpolation=interp),0,1)\n",
        "  return im[(im.shape[0]-size[0])//2:(im.shape[0]-size[0])//2+size[0], (im.shape[1]-size[1])//2:(im.shape[1]-size[1])//2+size[1]]\n",
        " \n",
        "source_image = imageio.imread('/content/image')\n",
        "source_image = pad_crop_resize(source_image, *get_crop(source_image, center_body=center_image_to_body, crop_body=crop_image_to_body, expansion_factor=image_crop_expansion_factor))\n",
        "imageio.imwrite('/content/crop.png', (source_image*255).astype(np.uint8))\n",
        "\n",
        "if os.path.exists('/content/background'):\n",
        "  bg_image = imageio.imread('/content/background')\n",
        "  bg_image = crop_resize(bg_image/255, source_image.shape[:2], crop=keep_aspect_background)\n",
        "  imageio.imwrite('/content/bg_crop.png', (bg_image*255).astype(np.uint8))\n",
        "\n",
        "#shutil.rmtree('/content/images', ignore_errors=True)\n",
        "#os.makedirs('/content/images')\n",
        "with imageio.get_reader('/content/video', format='mp4') as reader:\n",
        "  fps = reader.get_meta_data()['fps']\n",
        "''' \n",
        "  driving_video = []\n",
        "  rects = None\n",
        "  try:\n",
        "      for i,im in enumerate(reader):\n",
        "          if not crop_video_to_body:\n",
        "              break\n",
        "          rects, _ = hog.detectMultiScale(im, winStride=(4, 4),padding=(8,8), scale=video_crop_expansion_factor)\n",
        "          if rects is not None and len(rects):\n",
        "              break\n",
        "      x0,x1,y0,y1 = get_crop(im, center_body=center_video_to_body, crop_body=crop_video_to_body, expansion_factor=video_crop_expansion_factor, rects=rects)\n",
        "      reader.set_image_index(0)\n",
        "      for j,im in enumerate(reader):\n",
        "          vid_frame = pad_crop_resize(im,x0,x1,y0,y1)\n",
        "          #driving_video.append(vid_frame)\n",
        "          imageio.imwrite(os.path.join('/content/images','%05d.jpg'%j), (vid_frame*255).astype(np.uint8))\n",
        "  except RuntimeError:\n",
        "      pass\n",
        " \n",
        "def vid_display(source, driving, generated=None):\n",
        "    fig = plt.figure(figsize=(8 + 4 * (generated is not None), 6))\n",
        " \n",
        "    ims = []\n",
        "    for i in range(len(driving)):\n",
        "        cols = [source]\n",
        "        cols.append(driving[i])\n",
        "        if generated is not None:\n",
        "            cols.append(generated[i])\n",
        "        im = plt.imshow(np.concatenate(cols, axis=1), animated=True)\n",
        "        plt.axis('off')\n",
        "        ims.append([im])\n",
        " \n",
        "    ani = animation.ArtistAnimation(fig, ims, interval=50, repeat_delay=1000)\n",
        "    plt.close()\n",
        "    return ani\n",
        " \n",
        "clear_output()\n",
        "if rects is not None and len(rects):\n",
        "    print('first found body in frame %d'%i)\n",
        "print('number of frames: %d'%j)\n",
        "HTML(vid_display(source_image, driving_video).to_html5_video())\n",
        "'''\n",
        "pass"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BJskZDTiEKMB",
        "cellView": "form"
      },
      "source": [
        "#@title Animate\n",
        "#@markdown 1. The 'pose_fc' is the smooth factor of the temporal poses. The smaller of this value, the smoother of the temporal poses.\n",
        "#@markdown 2. The 'cam_fc' is the smooth factor of the temporal cameras (locations in the image space). The smaller of this value, the smoother of the locations in sequences.\n",
        "\n",
        "pose_fc = 300 #@param {type:'integer'}\n",
        "cam_fc = 100 #@param {type:'integer'}\n",
        "background = 'replace' #@param ['none', 'inpaint', 'replace']\n",
        "\n",
        "%cd /content/iPERCore\n",
        "\n",
        "import os.path as osp\n",
        "import platform\n",
        "import argparse\n",
        "import time\n",
        "import sys\n",
        "import subprocess\n",
        "from IPython.display import HTML\n",
        "from base64 import b64encode\n",
        "\n",
        "# the gpu ids\n",
        "gpu_ids = \"0\"\n",
        "\n",
        "# the default number of source images, it will be updated if the actual number of sources <= num_source\n",
        "num_source = 1\n",
        "\n",
        "# the assets directory. This is very important, please download it from `one_drive_url` firstly.\n",
        "assets_dir = \"/content/iPERCore/assets\"\n",
        "\n",
        "# the output directory.\n",
        "output_dir = \"./results\"\n",
        "shutil.rmtree(output_dir, ignore_errors=True)\n",
        "\n",
        "# symlink from the actual assets directory to this current directory\n",
        "work_asserts_dir = os.path.join(\"./assets\")\n",
        "if not os.path.exists(work_asserts_dir):\n",
        "    os.symlink(osp.abspath(assets_dir), osp.abspath(work_asserts_dir),\n",
        "               target_is_directory=(platform.system() == \"Windows\"))\n",
        "\n",
        "cfg_path = osp.join(work_asserts_dir, \"configs\", \"deploy.toml\")\n",
        "\n",
        "# This is a specific model name, and it will be used if you do not change it. This is the case of `trump`\n",
        "model_id = \"mymodel\"\n",
        "\n",
        "# the source input information, here \\\" is escape character of double duote \"\n",
        "src_path = \"\\\"path?=/content/crop.png,name?=mymodel\"\n",
        "if background=='replace' and os.path.exists('/content/bg_crop.png'):\n",
        "  src_path += ',bg_path?=/content/bg_crop.png'\n",
        "src_path += '\"'\n",
        "\n",
        "!cp /content/video /content/video.mp4\n",
        "ref_path = \"\\\"path?=/content/video.mp4,\"  \\\n",
        "             \"name?=myoutput,\" \\\n",
        "             \"pose_fc?=%d,\"\\\n",
        "             \"cam_fc?=%d,\"\\\n",
        "             \"fps?=%f\\\"\"%(pose_fc,cam_fc,fps)\n",
        "options = ''\n",
        "if background=='inpaint':\n",
        "  options += ' --use_inpaintor'\n",
        "!python -m iPERCore.services.run_imitator --gpu_ids $gpu_ids --num_source $num_source --image_size $image_size --output_dir $output_dir --model_id $model_id --cfg_path $cfg_path --src_path $src_path --ref_path $ref_path $options\n",
        "\n",
        "clear_output()\n",
        "mp4 = open(\"./results/primitives/mymodel/synthesis/imitations/mymodel-myoutput.mp4\", \"rb\").read()\n",
        "data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n",
        "HTML(f\"\"\"\n",
        "<video width=\"100%\" height=\"100%\" controls autoplay loop>\n",
        "      <source src=\"{data_url}\" type=\"video/mp4\">\n",
        "</video>\"\"\")\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "NCU7PkLNrLOm"
      },
      "source": [
        "#@title Download\n",
        "#@markdown 1. If it fails try running this cell again.\n",
        "#@markdown 2. Alternatively, you can manually download \"output.mp4\", \"combined.mp4\" from the folder on the left (click \"Refresh\" if missing).\n",
        "\n",
        "print() #see https://github.com/googlecolab/colabtools/issues/468\n",
        "from google.colab import files\n",
        "files.download('./results/primitives/mymodel/synthesis/imitations/mymodel-myoutput.mp4') #fails for Firefox private window"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}
